{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End-to-End FINN Flow for a Simple Convolutional Net\n",
    "-----------------------------------------------------------------\n",
    "\n",
    "In this notebook, we will go through the FINN steps needed to take a binarized convolutional network all the way down to a heterogeneous streaming dataflow accelerator running on the FPGA. \n",
    "\n",
    "It's recommended to go through the simpler [end-to-end notebook for a fully connected network](tfc_end2end_example.ipynb) first, since many steps here are very similar and we will focus on what is done differently for convolutions.\n",
    "\n",
    "This notebook is quite lengthy, and some of the cells (involving Vivado synthesis) may take up to an hour to finish running. To let you save and resume your progress, we will save the intermediate ONNX models that are generated in the various steps to disk, so that you can jump back directly to where you left off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Introduction to the CNV-w1a1 Network\n",
    "\n",
    "The particular quantized neural network (QNN) we will be targeting in this notebook is referred to as CNV-w1a1 and it classifies 32x32 RGB images into one of ten CIFAR-10 classes. All weights and activations in this network are quantized to bipolar values (either -1 or +1), with the exception of the input (which is RGB with 8 bits per channel) and the final output (which is 32-bit numbers). It first appeared in the original [FINN paper](https://arxiv.org/abs/1612.07119) from ISFPGA'17 with the name CNV, as a variant of the binarized convolutional network from the [BinaryNet paper](https://arxiv.org/abs/1602.02830), in turn inspired by the VGG-11 topology which was the runner-up for the 2014 [ImageNet Large Scale Visual Recognition Challenge](http://www.image-net.org/challenges/LSVRC/).\n",
    "\n",
    "\n",
    "You'll have a chance to interactively examine the layers that make up the network in Netron in a moment, so that's enough about the network for now. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Recap of the End-to-End Flow\n",
    "\n",
    "The FINN compiler comes with many *transformations* that modify the ONNX representation of the network according to certain patterns. This notebook will demonstrate a *possible* sequence of such transformations to take a particular trained network all the way down to hardware, as shown in the figure below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](finn-design-flow-example.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) to bring the network into a form in which each layer can be represented by either a Vitis HLS function or a Verilog module. The model then gets passed to Vivado IPI stitching (orange section), and finally a PYNQ overlay bitfile is built and can be tested on a PYNQ board (yellow section).\n",
    "There is an additional section for functional verification (red section) on the right side of the diagram, which we will not cover in this notebook. For details please take a look in the verification notebook which you can find [here](tfc_end2end_verification.ipynb)\n",
    "\n",
    "\n",
    "We will use the helper function `showInNetron` to show the ONNX model at the current transformation step. The Netron displays are interactive, but they only work when running the notebook actively and not on GitHub (i.e. if you are viewing this on GitHub you'll only see blank squares)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.util.basic import make_build_dir\n",
    "from finn.util.visualization import showInNetron\n",
    "import os\n",
    "    \n",
    "build_dir = os.environ[\"FINN_BUILD_DIR\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Brevitas Export, FINN Import and Tidy-Up\n",
    "\n",
    "Similar to what we did in the TFC-w1a1 end-to-end notebook, we will start by exporting the [pretrained CNV-w1a1 network](https://github.com/Xilinx/brevitas/tree/master/src/brevitas_examples/bnn_pynq) to ONNX, importing that into FINN and running the \"tidy-up\" transformations to have a first look at the topology. The network will be exported in QONNX format and then converted into the FINN-ONNX format to prepare it for the FINN compiler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import onnx\n",
    "from finn.util.test import get_test_model_trained\n",
    "from brevitas.export import export_qonnx\n",
    "from qonnx.util.cleanup import cleanup as qonnx_cleanup\n",
    "from qonnx.core.modelwrapper import ModelWrapper\n",
    "from finn.transformation.qonnx.convert_qonnx_to_finn import ConvertQONNXtoFINN\n",
    "from qonnx.transformation.infer_shapes import InferShapes\n",
    "from qonnx.transformation.fold_constants import FoldConstants\n",
    "from qonnx.transformation.general import GiveReadableTensorNames, GiveUniqueNodeNames, RemoveStaticGraphInputs\n",
    "\n",
    "cnv = get_test_model_trained(\"CNV\", 1, 1)\n",
    "export_onnx_path = build_dir + \"/end2end_cnv_w1a1_export.onnx\"\n",
    "export_qonnx(cnv, torch.randn(1, 3, 32, 32), export_onnx_path)\n",
    "qonnx_cleanup(export_onnx_path, out_file=export_onnx_path)\n",
    "model = ModelWrapper(export_onnx_path)\n",
    "model = model.transform(ConvertQONNXtoFINN())\n",
    "model = model.transform(InferShapes())\n",
    "model = model.transform(FoldConstants())\n",
    "model = model.transform(GiveUniqueNodeNames())\n",
    "model = model.transform(GiveReadableTensorNames())\n",
    "model = model.transform(RemoveStaticGraphInputs())\n",
    "model.save(build_dir + \"/end2end_cnv_w1a1_tidy.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the model is exported, let's have a look at its layer structure with Netron. Remember that the visualization below is interactive, you can click on the individual nodes and view the layer attributes, trained weights and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showInNetron(build_dir+\"/end2end_cnv_w1a1_tidy.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the network is composed of a repeating convolution-convolution-maxpool layer pattern to extract features using 3x3 convolution kernels (with weights binarized), followed by fully connected layers acting as the classifier. Also notice the initial `MultiThreshold` layer at the beginning of the network, which is quantizing float inputs to 8-bit ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Pre- and Postprocessing <a id='prepost'></a>\n",
    "\n",
    "Preprocessing and postprocessing steps can be added directly in the ONNX graph. In this case, the preprocessing step divides the input `uint8` data by 255 so the inputs to the CNV-w1a1 network are bounded between [0, 1]. The postprocessing step takes the output of the network and returns the index (0-9) of the image category with the highest probability (top-1). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.util.pytorch import ToTensor\n",
    "from qonnx.transformation.merge_onnx_models import MergeONNXModels\n",
    "from qonnx.core.datatype import DataType\n",
    "\n",
    "model = ModelWrapper(build_dir+\"/end2end_cnv_w1a1_tidy.onnx\")\n",
    "global_inp_name = model.graph.input[0].name\n",
    "ishape = model.get_tensor_shape(global_inp_name)\n",
    "# preprocessing: torchvision's ToTensor divides uint8 inputs by 255\n",
    "totensor_pyt = ToTensor()\n",
    "chkpt_preproc_name = build_dir+\"/end2end_cnv_w1a1_preproc.onnx\"\n",
    "export_qonnx(totensor_pyt, torch.randn(ishape), chkpt_preproc_name)\n",
    "qonnx_cleanup(chkpt_preproc_name, out_file=chkpt_preproc_name)\n",
    "pre_model = ModelWrapper(chkpt_preproc_name)\n",
    "pre_model = pre_model.transform(ConvertQONNXtoFINN())\n",
    "\n",
    "# join preprocessing and core model\n",
    "model = model.transform(MergeONNXModels(pre_model))\n",
    "# add input quantization annotation: UINT8 for all BNN-PYNQ models\n",
    "global_inp_name = model.graph.input[0].name\n",
    "model.set_tensor_datatype(global_inp_name, DataType[\"UINT8\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from qonnx.transformation.insert_topk import InsertTopK\n",
    "from qonnx.transformation.infer_datatypes import InferDataTypes\n",
    "\n",
    "# postprocessing: insert Top-1 node at the end\n",
    "model = model.transform(InsertTopK(k=1))\n",
    "chkpt_name = build_dir+\"/end2end_cnv_w1a1_pre_post.onnx\"\n",
    "# tidy-up again\n",
    "model = model.transform(InferShapes())\n",
    "model = model.transform(FoldConstants())\n",
    "model = model.transform(GiveUniqueNodeNames())\n",
    "model = model.transform(GiveReadableTensorNames())\n",
    "model = model.transform(InferDataTypes())\n",
    "model = model.transform(RemoveStaticGraphInputs())\n",
    "model.save(chkpt_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showInNetron(build_dir+\"/end2end_cnv_w1a1_pre_post.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. How FINN Implements Convolutions: Lowering and Streamlining\n",
    "\n",
    "In FINN, we implement convolutions with the *lowering* approach: we convert them to matrix-matrix multiply operations, where one of the matrices is generated by sliding a window over the input image. You can read more about the sliding window operator and how convolution lowering works [in this notebook](https://github.com/maltanar/qnn-inference-examples/blob/master/3-convolutional-binarized-gtsrb.ipynb). The streaming dataflow architecture we will end up with is going to look something like this figure from the [FINN-R paper](https://arxiv.org/abs/1809.04570):\n",
    "\n",
    "![](cnv-mp-fc.png)\n",
    "\n",
    "Note how the convolution layer looks very similar to the fully connected one in terms of the matrix-vector-threshold unit (MVTU) or sometimes called matrix-vector-activation unit (MVAU). But now the MVTU is preceded by a sliding window unit that produces the matrix from the input image. All of these building blocks, including the `MaxPool` layer you see in this figure, exist as templated Vitis HLS C++ functions in [finn-hlslib](https://github.com/Xilinx/finn-hlslib) and/or as RTL modules in [finn-rtllib](https://github.com/Xilinx/finn/tree/main/finn-rtllib).\n",
    "\n",
    "\n",
    "To target this kind of hardware architecture with our network we'll apply a convolution lowering transformation, in addition to streamlining. You may recall the *streamlining transformation* that we applied to the TFC-w1a1 network, which is a series of mathematical simplifications that allow us to get rid of floating point scaling operations by implementing few-bit activations as thresholding operations. \n",
    "\n",
    "**The current implementation of streamlining is highly network-specific and may not work for your network if its topology is very different than the example network here. We hope to rectify this in future releases.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.streamline import Streamline\n",
    "from qonnx.transformation.lower_convs_to_matmul import LowerConvsToMatMul\n",
    "from qonnx.transformation.bipolar_to_xnor import ConvertBipolarMatMulToXnorPopcount\n",
    "import finn.transformation.streamline.absorb as absorb\n",
    "from finn.transformation.streamline.reorder import MakeMaxPoolNHWC, MoveScalarLinearPastInvariants\n",
    "from qonnx.transformation.infer_data_layouts import InferDataLayouts\n",
    "from qonnx.transformation.general import RemoveUnusedTensors\n",
    "\n",
    "model = ModelWrapper(build_dir + \"/end2end_cnv_w1a1_pre_post.onnx\")\n",
    "model = model.transform(MoveScalarLinearPastInvariants())\n",
    "model = model.transform(Streamline())\n",
    "model = model.transform(LowerConvsToMatMul())\n",
    "model = model.transform(MakeMaxPoolNHWC())\n",
    "model = model.transform(absorb.AbsorbTransposeIntoMultiThreshold())\n",
    "model = model.transform(ConvertBipolarMatMulToXnorPopcount())\n",
    "model = model.transform(Streamline())\n",
    "# absorb final add-mul nodes into TopK\n",
    "model = model.transform(absorb.AbsorbScalarMulAddIntoTopK())\n",
    "model = model.transform(InferDataLayouts())\n",
    "model = model.transform(RemoveUnusedTensors())\n",
    "model.save(build_dir + \"/end2end_cnv_w1a1_streamlined.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't go into too much detail about what happens in each transformation and why they are called in the particular order they are (feel free to visualize the intermediate steps using Netron yourself if you are curious) but here is a brief summary:\n",
    "\n",
    "* `Streamline` moves floating point scaling and addition operations closer to the input of the nearest thresholding activation and absorbs them into thresholds\n",
    "* `LowerConvsToMatMul` converts ONNX `Conv` nodes into sequences of `Im2Col, MatMul` nodes as discussed above. `Im2Col` is a custom FINN ONNX high-level node type that implements the sliding window operator.\n",
    "* `MakeMaxPoolNHWC` and `AbsorbTransposeIntoMultiThreshold` convert the *data layout* of the network into the NHWC data layout that finn-hlslib and finn-rtllib primitives use. NCHW means the tensor dimensions are ordered as `(N : batch, H : height, W : width, C : channels)` (assuming 2D images). The ONNX standard ops normally use the NCHW layout, but the ONNX intermediate representation itself does not dictate any data layout.\n",
    "* You may recall `ConvertBipolarMatMulToXnorPopcount` from the TFC-w1a1 example, which is needed to implement bipolar-by-bipolar (w1a1) networks correctly using finn-hlslib.\n",
    "\n",
    "Let's visualize the streamlined and lowered network with Netron. Observe how all the `Conv` nodes have turned into pairs of `Im2Col, MatMul` nodes, and many nodes including `BatchNorm, Mul, Add` nodes have disappeared and replaced with `MultiThreshold` nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showInNetron(build_dir+\"/end2end_cnv_w1a1_streamlined.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Partitioning, Conversion to HW Layers and Folding\n",
    "\n",
    "The next steps will be (again) very similar to what we did for the TFC-w1a1 network. We'll first convert the layers that we can put into the FPGA into their HW equivalents, separate them out into a *dataflow partition* and specialize them to HLS variants:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.util.basic import pynq_part_map\n",
    "# change this if you have a different PYNQ board, see list above\n",
    "pynq_board = \"Pynq-Z1\"\n",
    "fpga_part = pynq_part_map[pynq_board]\n",
    "target_clk_ns = 10\n",
    "\n",
    "import finn.transformation.fpgadataflow.convert_to_hw_layers as to_hw\n",
    "from finn.transformation.fpgadataflow.create_dataflow_partition import (\n",
    "    CreateDataflowPartition,\n",
    ")\n",
    "from finn.transformation.move_reshape import RemoveCNVtoFCFlatten\n",
    "from finn.transformation.fpgadataflow.specialize_layers import SpecializeLayers\n",
    "from qonnx.custom_op.registry import getCustomOp\n",
    "from qonnx.transformation.infer_data_layouts import InferDataLayouts\n",
    "\n",
    "model = ModelWrapper(build_dir + \"/end2end_cnv_w1a1_streamlined.onnx\")\n",
    "model = model.transform(to_hw.InferBinaryMatrixVectorActivation())\n",
    "model = model.transform(to_hw.InferQuantizedMatrixVectorActivation())\n",
    "# TopK to LabelSelect\n",
    "model = model.transform(to_hw.InferLabelSelectLayer())\n",
    "# input quantization (if any) to standalone thresholding\n",
    "model = model.transform(to_hw.InferThresholdingLayer())\n",
    "model = model.transform(to_hw.InferConvInpGen())\n",
    "model = model.transform(to_hw.InferStreamingMaxPool())\n",
    "# get rid of Reshape(-1, 1) operation between hw nodes\n",
    "model = model.transform(RemoveCNVtoFCFlatten())\n",
    "# get rid of Tranpose -> Tranpose identity seq\n",
    "model = model.transform(absorb.AbsorbConsecutiveTransposes())\n",
    "# infer tensor data layouts\n",
    "model = model.transform(InferDataLayouts())\n",
    "parent_model = model.transform(CreateDataflowPartition())\n",
    "parent_model.save(build_dir + \"/end2end_cnv_w1a1_dataflow_parent.onnx\")\n",
    "sdp_node = parent_model.get_nodes_by_op_type(\"StreamingDataflowPartition\")[0]\n",
    "sdp_node = getCustomOp(sdp_node)\n",
    "dataflow_model_filename = sdp_node.get_nodeattr(\"model\")\n",
    "# save the dataflow partition with a different name for easier access\n",
    "# and specialize the layers to HLS variants\n",
    "dataflow_model = ModelWrapper(dataflow_model_filename)\n",
    "dataflow_model = dataflow_model.transform(SpecializeLayers(fpga_part))\n",
    "dataflow_model.save(build_dir + \"/end2end_cnv_w1a1_dataflow_model.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the additional `RemoveCNVtoFCFlatten` transformation that was not used for TFC-w1a1. In the last Netron visualization you may have noticed a `Reshape` operation towards the end of the network where the convolutional part of the network ends and the fully-connected layers started. That `Reshape` is essentialy a tensor flattening operation, which we can remove for the purposes of hardware implementation. We can examine the contents of the dataflow partition with Netron, and observe the `ConvolutionInputGenerator`, `MatrixVectorActivation` and `StreamingMaxPool_Batch` nodes that implement the sliding window, matrix multiply and maxpool operations. *Note that the MatrixVectorActivation instances following the ConvolutionInputGenerator nodes are really implementing the convolutions, despite the name. The final three MatrixVectorActivation instances implement actual FC layers.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showInNetron(build_dir + \"/end2end_cnv_w1a1_dataflow_parent.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that pretty much everything has gone into the `StreamingDataflowPartition` node; the only operation remaining is to apply a `Transpose` to obtain NHWC input from a NCHW input (the ONNX default). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showInNetron(build_dir + \"/end2end_cnv_w1a1_dataflow_model.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have to set the *folding factors* for certain layers to adjust the performance of our accelerator, similar to the TFC-w1a1 example. We'll also set the desired FIFO depths around those layers, which are important to achieve full throughput in the accelerator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ModelWrapper(build_dir + \"/end2end_cnv_w1a1_dataflow_model.onnx\")\n",
    "fc_layers = model.get_nodes_by_op_type(\"MVAU_hls\")\n",
    "# each tuple is (PE, SIMD, in_fifo_depth) for a layer\n",
    "folding = [\n",
    "    (16, 3, [128]),\n",
    "    (32, 32, [128]),\n",
    "    (16, 32, [128]),\n",
    "    (16, 32, [128]),\n",
    "    (4, 32, [81]),\n",
    "    (1, 32, [2]),\n",
    "    (1, 4, [2]),\n",
    "    (1, 8, [128]),\n",
    "    (5, 1, [3]),\n",
    "]\n",
    "for fcl, (pe, simd, ififodepth) in zip(fc_layers, folding):\n",
    "    fcl_inst = getCustomOp(fcl)\n",
    "    fcl_inst.set_nodeattr(\"PE\", pe)\n",
    "    fcl_inst.set_nodeattr(\"SIMD\", simd)\n",
    "    fcl_inst.set_nodeattr(\"inFIFODepths\", ififodepth)\n",
    "\n",
    "# use same SIMD values for the sliding window operators\n",
    "swg_layers = model.get_nodes_by_op_type(\"ConvolutionInputGenerator_rtl\")\n",
    "for i in range(len(swg_layers)):\n",
    "    swg_inst = getCustomOp(swg_layers[i])\n",
    "    simd = folding[i][1]\n",
    "    swg_inst.set_nodeattr(\"SIMD\", simd)\n",
    "\n",
    "model = model.transform(GiveUniqueNodeNames())\n",
    "model.save(build_dir + \"/end2end_cnv_w1a1_folded.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we visualize in Netron to observe the folding factors in the `PE` and `SIMD` attributes of each `MVAU_hls`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "showInNetron(build_dir + \"/end2end_cnv_w1a1_folded.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our network is now ready and we can start with the hardware generation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Hardware Generation\n",
    "\n",
    "From this point onward, the steps we have to follow do not depend on the particular network and will be exactly the same as the TFC-w1a1 example. **which may take about 30 minutes depending on your host computer**. For more details about what's going on in this step, please consult the [TFC end-to-end notebook](tfc_end2end_example.ipynb) or the appropriate section in the [FINN documentation](https://finn.readthedocs.io/en/latest/hw_build.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.fpgadataflow.make_zynq_proj import ZynqBuild\n",
    "model = ModelWrapper(build_dir+\"/end2end_cnv_w1a1_folded.onnx\")\n",
    "model = model.transform(ZynqBuild(platform = pynq_board, period_ns = target_clk_ns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the `ZynqBuild` we run one additional transformation to generate a PYNQ driver for the accelerator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.fpgadataflow.make_pynq_driver import MakePYNQDriver\n",
    "model = model.transform(MakePYNQDriver(\"zynq-iodma\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(build_dir + \"/end2end_cnv_w1a1_synth.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Deployment and Execution\n",
    "\n",
    "The bitfile and generated driver files(s) will be copied into a deployment folder which then can be used to run the network on the PYNQ board."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shutil import copy\n",
    "from distutils.dir_util import copy_tree\n",
    "\n",
    "# create directory for deployment files\n",
    "deployment_dir = make_build_dir(prefix=\"pynq_deployment_\")\n",
    "model.set_metadata_prop(\"pynq_deployment_dir\", deployment_dir)\n",
    "\n",
    "# get and copy necessary files\n",
    "# .bit and .hwh file\n",
    "bitfile = model.get_metadata_prop(\"bitfile\")\n",
    "hwh_file = model.get_metadata_prop(\"hw_handoff\")\n",
    "deploy_files = [bitfile, hwh_file]\n",
    "\n",
    "for dfile in deploy_files:\n",
    "    if dfile is not None:\n",
    "        copy(dfile, deployment_dir)\n",
    "\n",
    "# driver.py and python libraries\n",
    "pynq_driver_dir = model.get_metadata_prop(\"pynq_driver_dir\")\n",
    "copy_tree(pynq_driver_dir, deployment_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next to these files, we will also need an example numpy array to test the network on the PYNQ board. (*and before you ask, that's supposed to be a cat (CIFAR-10 class number 3)*) Recall that we partitioned our original network into a parent graph that contained the non-synthesizable nodes and a child graph that contained the bulk of the network, which we turned into a bitfile. The only operator left outside the FPGA partition was a `Transpose` to convert NCHW images into NHWC ones. Thus, we can skip the execution in the parent as long as we ensure our image has the expected data layout. The example numpy array can then be saved as .npy file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import importlib_resources\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "ref = importlib_resources.files(\"finn.qnn-data\") / \"cifar10/cifar10-test-data-class3.npz\"\n",
    "with importlib_resources.as_file(ref) as fn:\n",
    "    x = np.load(fn)[\"arr_0\"]\n",
    "x = x.reshape(3, 32,32).transpose(1, 2, 0)\n",
    "plt.imshow(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ModelWrapper(build_dir + \"/end2end_cnv_w1a1_synth.onnx\")\n",
    "iname = model.graph.input[0].name\n",
    "ishape = model.get_tensor_shape(iname)\n",
    "np.save(deployment_dir + \"/input.npy\", x.reshape(ishape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ls {deployment_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shutil import make_archive\n",
    "make_archive('deploy-on-pynq-cnv', 'zip', deployment_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now download the created zipfile (File -> Open, mark the checkbox next to the deploy-on-pynq-tfc.zip and select Download from the toolbar), then copy it to your PYNQ board (for instance via scp or rsync). Then, run the following commands on the PYNQ board to extract the archive and run the execution:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```shell\n",
    "unzip deploy-on-pynq-cnv.zip -d finn-cnv-demo\n",
    "cd finn-cnv-demo\n",
    "sudo python3 -m pip install bitstring\n",
    "sudo python3 driver.py --exec_mode=execute --batchsize=1 --bitfile=resizer.bit --inputfile=input.npy\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output will be saved on the PYNQ board as `output.npy` and can be copied to the host and opened with `np.load()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validating the Accuracy on a PYNQ Board <a id='validation'></a>\n",
    "\n",
    "All the command line prompts here are meant to be executed with `sudo` on the PYNQ board.\n",
    "\n",
    "**Ensure that your PYNQ board has a working internet connecting for the next steps, since some there is some downloading involved.**\n",
    "\n",
    "To validate the accuracy, we first need to install the [`dataset-loading`](https://github.com/fbcotter/dataset_loading) Python package to the PYNQ board. This will give us a convenient way of downloading and accessing the MNIST dataset.\n",
    "\n",
    "\n",
    "Command to execute on PYNQ:\n",
    "\n",
    "```shell\n",
    "sudo pip3 install git+https://github.com/fbcotter/dataset_loading.git@0.0.4#egg=dataset_loading\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the `validate.py` script that was generated together with the driver to measure top-1 accuracy on the CIFAR-10 dataset.\n",
    "\n",
    "Command to execute on PYNQ:\n",
    "\n",
    "```shell\n",
    "sudo python3 validate.py --dataset cifar10 --batchsize 1000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the final top-1 accuracy is 84.19%, which is very close to the 84.22% reported on the [BNN-PYNQ accuracy table in Brevitas](https://github.com/Xilinx/brevitas/tree/master/src/brevitas_examples/bnn_pynq). "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
